1. Field of the Invention
The present invention relates to a system and method of providing input to a computing device and more specifically to translating data such as a speech utterance that are not within a grammar into translated data that has a meaning representation within the grammar.
2. Introduction
The present invention relates to interpreting data input to a computing device. The data may be in the form of speech, graffiti, motion and/or any combination of such input. The plurality of available input modes of input places this in the context of “multimodal” interaction with computing devices. Multimodal interfaces allow input and/or output to be conveyed over multiple channels such as speech, graphics and gesture. Multimodal grammars provide an expressive mechanism for quickly creating language processing capabilities for multimodal interfaces supporting input modes such as speech and pen. They support composite multimodal inputs by aligning speech input (words) and gesture input (represented as sequence of gesture symbols) while expressing the relation between the speech and gesture input and their combined semantic representation. In Johnston and Bangalore, “Finite-state multimodal parsing and understanding,” Proceedings of COLING, Saarbrucken, Germany, 2000, pp. 369-375 and Johnston and Bangalore, “Finite state multimodal integration and understanding,” Journal of Natural Language Engineering, vol. 11, no. 2, pp. 159-187, 2005 (both articles incorporated herein by reference) it has been shown that such grammars can be compiled into finite-state transducers enabling effective processing of lattice input from speech and gesture recognition and mutual compensation for errors and ambiguities. See also, Srinivas Bangalore and Michael Johnston, “Balancing data-driven and rule-based approaches in the context of a multimodal conversational system,” Proceedings of ICSLP, Beijing, China, 2000, pp. 126-129, incorporated herein by reference.
However, like other approaches based on hand-crafted grammars, multimodal grammars can be brittle with respect to extra-grammatical or erroneous input. For example, one limitation is that a language model directly projected from the speech portion of a hand-crafted multimodal grammar will not be able to recognize any strings that are not accepted by the grammar. Therefore, what is needed in the art is an improved method of interpreting and processing received input to a computing device where the input is not in the form accepted by a grammar.